Image processing device, stereo camera device, vehicle, and image processing method

ABSTRACT

Provided is an image processing device including: an input interface, which is configured to acquire a first captured image and a second captured image that are captured by a plurality of imaging units and; and a controller, which is configured to calculate parallax by performing a one-dimensional matching based on pixel values of the first captured image and pixel values of the second captured image, extracts one or more first feature points from a region in the first captured image that includes continuous pixels having a difference in parallax which is within a predetermined range, extract one or more second feature points corresponding respectively to the first feature points by performing a two-dimensional matching with the first feature points, and calibrate the imaging unit based on positions of the first feature points and positions of the second feature points.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority from and the benefit of Japanese PatentApplication No. 2015-126817 filed on Jun. 24, 2015, the entire contentsof which are incorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates to an image processing device, a stereocamera device, a vehicle, and an image processing method.

BACKGROUND

In recent years, stereo cameras that simultaneously capture images of atarget, such as an object or a human, using a plurality of cameras, andthat use the captured images to measure a distance to the target basedon the triangulation principle, are known. Such a stereo cameras may bemounted, for example, to a vehicle to inform a driver of the presence ofa target that is present in the vicinity of the vehicle and to assistsafe driving.

SUMMARY

One of aspects of the present disclosure resides in an image processingdevice that is configured to calibrate a stereo camera including aplurality of imaging units and that includes an input interface and acontroller. The input interface is configured to acquire a firstcaptured image and a second captured image captured by the plurality ofimaging units. The controller is configured to calculate parallax byperforming a one-dimensional matching based on pixel values of the firstcaptured image and pixel values of the second captured image. Thecontroller is also configured to extract one or more first featurepoints from a region in the first captured image that includescontinuous pixels having a difference in parallax which is within apredetermined range and extract one or more second feature pointscorresponding respectively to the one or more first feature points byperforming a two-dimensional matching between the one or more firstfeature points and pixels in the second captured image. The controlleris also configured to calibrate at least one of the plurality of imagingunits based on positions of the one or more first feature points andpositions of the one or more second feature points.

Another aspect of the present disclosure resides in a stereo cameradevice, including: a plurality of imaging units; and an image processingdevice that is configured to calibrate a stereo camera including theplurality of imaging units. The image processing device includes aninput interface and a controller. The input interface is configured toreceive input of a first captured image and a second captured imagecaptured by the plurality of imaging units. The controller is configuredto calculate parallax by performing a one-dimensional matching based onpixel values of the first captured image and pixel values of the secondcaptured image. The controller is also configured to extract one or morefirst feature points from a region in the first captured image thatincludes continuous pixels having a difference in parallax which iswithin a predetermined range and extract one or more second featurepoints corresponding respectively to the one or more first featurepoints by performing a two-dimensional matching between the one or morefirst feature points and pixels in the second captured image. Thecontroller is also configured to calibrate at least one of the pluralityof imaging units based on positions of the one or more first featurepoints and positions of the one or more second feature points.

Yet another aspect of the present disclosure resides in a vehicle,including a stereo camera device. The stereo camera device includes: aplurality of imaging units; and an image processing device that isconfigured to calibrate a stereo camera including the plurality ofimaging units. The image processing device includes an input interfaceand a controller. The input interface is configured to receive input ofa first captured image and a second captured image captured by theplurality of imaging units. The controller is configured to calculateparallax by performing a one-dimensional matching based on pixel valuesof the first captured image and pixel values of the second capturedimage. The controller is also configured to extract one or more firstfeature points from a region in the first captured image that includescontinuous pixels having a difference in parallax which is within apredetermined range and extract one or more second feature pointscorresponding respectively to the one or more first feature points byperforming a two-dimensional matching between the one or more featurepoints and pixels in the second captured image. The controller is alsoconfigured to calibrate at least one of the plurality of imaging unitsbased on positions of the one or more first feature points and positionsof the one or more second feature points.

Yet another aspect of the present disclosure resides in an imageprocessing method performed by an image processing device configured tocalibrate a stereo camera including a plurality of imaging units, thatis, the image processing method, in which a controller in the imageprocessing device performs steps including calculating parallax byperforming a one-dimensional matching based on pixel values of a firstcaptured image captured by a first imaging unit in a plurality ofimaging units and pixel values in a second captured image captured by asecond imaging unit in the plurality of imaging units that is differentfrom the first imaging unit, extracting one or more first feature pointsfrom a region in the first captured image that includes continuouspixels having a difference in parallax which is within a predeterminedrange, extracting one or more second feature points correspondingrespectively to the one or more first feature points by performing atwo-dimensional matching between the one or more first feature pointsand pixels in the second captured image, and calibrating at least one ofthe plurality of imaging units based on positions of the one or morefirst feature points and positions of the one or more second featurepoints.

BRIEF DESCRIPTION OF THE DRAWINGS

In the accompanying drawings:

FIG. 1 is a perspective view of a stereo camera device including animage processing device according to an embodiment;

FIG. 2 is a side view of a vehicle in which the stereo camera deviceillustrated in FIG. 1 is mounted;

FIG. 3 is a configuration diagram of the stereo camera deviceillustrated in FIG. 1;

FIGS. 4(1) to 4(4) illustrate a first captured image and a secondcaptured image, and one-dimensional pixel value distributions thereof,captured respectively by a first imaging unit and a second imaging unitillustrated in FIG. 1;

FIG. 5 is a conceptual view of parallax of pixels constituting a subjectregion of the first captured image illustrated in FIG. 4; and

FIG. 6 is a flowchart illustrating processing performed by the imageprocessing device illustrated in FIG. 1.

DETAILED DESCRIPTION

In order to accurately notify a user of the distance to a target in thevicinity thereof a stereo camera, must accurately measure a distancefrom a plurality of cameras to the target to be captured by theplurality of cameras. However, when the plurality of cameras is mountedat positions which deviate from a standard, the distance to the targetcannot be measured with accuracy.

Accordingly, an existing calibration method determines that a referenceregion is appropriate for matching and accurately calibrates thehorizontality of the camera when the maximum appearance frequency of thedistances in real space to the subject captured in the reference regionis greater than a predetermined value.

According to the above calibration method, since the reference regionappropriate for matching is determined based on the maximum appearancefrequency of the distances to the subject, a distance distributionhistogram needs to be generated, and this processing may require time.Furthermore, when positions in the reference region in which a portioncorresponding to the distance of the maximum appearance frequency iscaptured are discrete, occlusion may occur in the reference region, andthis makes accurate matching difficult.

An image processing device according to the present disclosurecalibrates an imaging unit by performing a one-dimensional matching tocalculate parallax and a two-dimensional matching with at least onefirst feature point in a first captured image captured by the stereocamera to extract a second feature point in a second captured image thatcorresponds to the first feature point.

The image processing device of the present disclosure is described indetail below with reference to the drawings.

As illustrated in FIG. 1, a stereo camera device 1 includes a pluralityof imaging units 11L and 11R having optical axes that are notoverlapping with each other, and an image processor 10. In thedescription below, the image processor 10 may also be called an imageprocessing device. Accordingly, a plurality of images capturedsimultaneously by the plurality of imaging units 11L and 11R will haveparallax d.

The term “simultaneous” as used herein is not limited to the exact sametime. For example, “simultaneous” imaging as used in the disclosureincludes (i) the plurality of cameras capturing images at the same time,(ii) the plurality of cameras capturing images in response to the samesignal, and (iii) the plurality of cameras capturing images at the sametime according to respective internal clocks. The imaging start time,the imaging end time, the transmission time of captured image data, andthe time at which another device receives image data are included in thetime standard for imaging. The stereo camera device 1 may include aplurality of cameras in a single housing. The stereo camera device 1 mayalso include two or more independent cameras positioned apart from eachother. The stereo camera device 1 is not limited to a plurality ofindependent cameras. In the present disclosure, a camera having anoptical mechanism that guides light incident at two separate locationsto one optical detector, for example, may be adopted as the stereocamera device 1.

The first imaging unit 11L and the second imaging unit 11R include asolid-state image sensor. A solid-state image sensor includesCharge-Coupled Device (CCD) image sensor and a Complementary MOS (CMOS)image sensor. The first imaging unit 11L and the second imaging unit 11Rmay include a lens mechanism. The first imaging unit 11L and the secondimaging unit 11R capture an image of real space to generate a firstcaptured image 14L and a second captured image 14R.

The plurality of imaging units 11L and 11R is mounted in a vehicle 15,which is placed on a horizontal plane as illustrated in FIG. 2, in amanner such that the optical axes of the imaging units 11L and 11R areparallel, lens surfaces and imaging surfaces of the imaging units 11Land 11R are on the same planes, and the baseline length direction ishorizontal. In the state where the two imaging units 11L and 11R aremounted in the correct position and correct posture, the stereo cameradevice 1 is able to measure a distance from the stereo camera device 1to the subject with accuracy. Hereinafter, one of the two imaging units11L and 11R that is mounted on the left side when looking at the stereocamera device 1 from the opposite side of the subject is called thefirst imaging unit 11L, and the other one that is mounted on the rightside is called the second imaging unit 11R.

In the present embodiment, the first imaging unit 11L and the secondimaging unit 11R are able to capture an image of the outside of thevehicle 15 via the windshield of the vehicle 15. In the presentembodiment, the first imaging unit 11L and the second imaging unit 11Rmay be fixed to any one of the front bumper, the fender grill, the sidefenders, the light modules, and the bonnet of the vehicle 15.

The term “parallel” as used above is not limited to strict parallelism.The term “parallel” may encompass a substantially parallel state inwhich the optical axes of the imaging units 11L and 11R are consideredsubstantially parallel, e.g., misaligned and not perfectly parallel. Theterm “horizontal” as used above is not limited to strict horizontality.The term “horizontal” may encompass a substantially horizontal state inwhich the baseline length direction is for example deviated from theperfectly horizontal position with respect to the direction of thehorizon plane.

Here, “vehicle” in the present disclosure includes, but is not limitedto, automobiles, railway vehicles, industrial vehicles, and vehicles fordaily life. For example, “vehicle” may include aircraft that travel downa runway. Automobiles include, but are not limited to, passengervehicles, trucks, buses, two-wheeled vehicles, and trolley buses, andmay include other vehicles that drive on a road. Railway vehiclesinclude, but are not limited to, locomotives, freight cars, passengercars, streetcars, guided railway vehicles, ropeways, cable cars, linearmotor cars, and monorails, and may include other vehicles that travelalong a rail. Industrial vehicles include industrial vehicles foragriculture and for construction. Industrial vehicles include, but arenot limited to, forklifts and golf carts. Industrial vehicles foragriculture include, but are not limited to, tractors, cultivators,translators, binders, combines, and lawnmowers. Industrial vehicles forconstruction include, but are not limited to, bulldozers, scrapers,backhoes, cranes, dump cars, and road rollers. Vehicles for daily lifeinclude, but are not limited to, bicycles, wheelchairs, baby carriages,wheelbarrows, and motorized, two-wheeled standing vehicles. Powerengines for the vehicle include, but are not limited to,internal-combustion engines including diesel engines, gasoline engines,and hydrogen engines, and electrical engines including motors. The“vehicle” is not limited to the above-listed types. For example,automobiles may include industrial vehicles that can drive on a road,and the same vehicle may be included in multiple categories.

Next, the image processor 10 is described with reference to FIG. 3. Asillustrated in FIG. 3, the image processor 10 includes an inputinterface 12 and a controller 13.

The input interface 12 is an input interface for inputting image data tothe image processor 10. A physical connector or a wireless communicationdevice may be used in the input interface 12. Physical connectorsinclude an electrical connector corresponding to transmission by anelectric signal, an optical connector corresponding to transmission byan optical signal, and an electromagnetic connector corresponding totransmission by an electromagnetic wave. Electrical connectors includeconnectors conforming to IEC60603, connectors conforming to the USBstandard, connectors comprising an RCA terminal, connectors comprisingan S terminal prescribed by EIAJ CP-1211A, connectors comprising a Dterminal prescribed by EIAJ RC-5237, connectors conforming to the HDMI®(HDMI is a registered trademark in Japan, other countries, or both)standard, and connector corresponding to a coaxial cable that includes aBNC terminal. Optical connectors include a variety of connectorsconforming to IEC 61754. Wireless communication devices include wirelesscommunication devices conforming to standards that include Bluetooth®(Bluetooth is a registered trademark in Japan, other countries, or both)and IEEE802.11. The wireless communication device includes at least oneantenna.

Image data of images captured by the first imaging unit 11L and thesecond imaging unit 11R is inputted to the input interface 12. The inputinterface 12 delivers the inputted image data to the controller 13.Input to the input interface 12 includes signals input over a wiredcable and signals input over a wireless connection. The input interface12 may correspond to the transmission method of an image signal in thestereo camera device 1.

The controller 13 includes one or a plurality of processors. Thecontroller 13 or the processors may include one or a plurality ofmemories that store programs for various processing and information onwhich operations are being performed. Memories include volatile andnonvolatile memories. Memories also include those independent ofprocessors and those embedded in processors. Processors includeuniversal processors that execute particular functions by readingparticular programs and dedicated processors that are specialized forparticular processing. Dedicated processors include an ApplicationSpecific Integrated Circuit (ASIC) for a specific application.Processors include a Programmable Logic Device (PLD). PLDs include aField-Programmable Gate Array (FPGA). The controller 13 may be either aSystem-on-a-Chip (SoC) or a System in a Package (SiP) with one processoror a plurality of processors that work together.

The controller 13 measures the distance in real space from the stereocamera device 1 to the subject captured in the first captured image 14Land the second captured image 14R, which have been inputted to the inputinterface 12.

In the stereo camera device 1, the controller 13 calculates the distancefrom the stereo camera device 1 to the subject in a spatial coordinatesystem as illustrated in FIG. 1. The spatial coordinate system includesan X-axis in the direction of the baseline length and a Y-axis and aZ-axis in two directions that are perpendicular to the baseline lengthand that are also perpendicular with respect to each other, with anypoint being defined as the origin. The optical axes of the first imagingunit 11L and the second imaging unit 11R are parallel to Z-axis, the rowdirection of the imaging surfaces is parallel to X-axis, and thecolumnar direction of the imaging surfaces is parallel to Y-axis. Therotation angle around X-axis is defined as pitch angle Φ, and therotation angle around Z-axis is defined as rho angle to in the spatialcoordinate system.

In the stereo camera device 1, both of the optical axes are parallel toZ-axis, and the columnar direction of the imaging surfaces is parallelto Y-axis, which is perpendicular to the baseline length direction.Accordingly, the positions of spot images of the same subject differonly in the row direction in the first captured image 14L and in thesecond captured image 14R. Accordingly, to perform calculation of thedistance at a high speed such as 30 fps, the stereo camera device 1performs one-dimensional matching along the direction parallel to thebaseline length, i.e., along the X-axis direction, to bring the spotimages of the same subject in the first captured image 14L and in thesecond captured image 14R into correspondence with each other.

However, the accuracy of the correspondence between the spot imagesaccording to the above one-dimensional matching decreases as adisplacement ΔY along the Y-axis direction of the first imaging unit 11Lwith respect to the second imaging unit 11R in external orientationparameters increases. Similarly, the accuracy of the correspondencebetween the spot images according to the one-dimensional matchingdecreases as misalignment Δϕ of pitch angle ϕ of the optical axesincreases. To address the above, as described below, the stereo cameradevice 1 performs calibration, for at least one of the position Y of theY-axis direction and pitch angle ϕ, the first imaging unit 11L, withreference to the second imaging unit 11R, based on the first capturedimage 14L and the second captured image 14R.

The controller 13 calculates parallax d of pixel positions in the firstcaptured image 14L and in the second captured image 14R. The pixelpositions in the first captured image 14L and in the second capturedimage 14R are represented by the image coordinate system (u, v), havinga U-axis that is parallel to the row direction of the imaging surfacesand a V-axis that is parallel to the columnar direction of the imagingsurfaces. The controller 13 calculates parallax d according to theone-dimensional matching along the row direction of the imagingsurfaces. In detail, the controller 13 compares one-dimensional pixelvalue distribution in the U-axis direction at different v-coordinates inthe first captured image 14L and one-dimensional pixel valuedistribution at the same v-coordinates in the second captured image 14R.The controller 13 calculates a difference in position of two pixelsincluding pixel values corresponding to each other in the twodistributions as parallax d.

Here, a detailed description of a method to calculate parallax d isprovided. The controller 13 determines a constant on V-axis for whichparallax d is to be calculated. Here, the controller 13 is assumed tocalculate parallax d for vi on the V-axis. The controller 13 extractspixel values of different pixels at v=v₁ from the first captured image14L and the second captured image 14R, as illustrated in FIGS. 4(1) and4(2). Pixel value distributions extracted from the first captured image14L and the second captured image 14R are illustrated for example inFIGS. 4(3) and 4(4). Based on the two extracted pixel valuedistributions, the controller 13 brings pixels of the second capturedimage 14R into correspondence with pixels of the first captured image14L according to the one-dimensional matching. That is, the controller13 extracts pixels on the second captured image 14R that are most likelyto represent the spot image formed by pixels on the first captured image14L and brings the extracted pixels into correspondence with the pixelson the first captured image 14L. The controller 13 calculates adifference between the position (u_(L1), v₁) and the position (u_(R1),v₁) of corresponding pixels respectively in the first captured image 14Land the second captured image 14R as parallax d=u_(L1)−u_(R1).

The controller 13 determines a region in a parallax image that includescontinuous pixels whose difference in parallax d between adjacent pixelsis within a predetermined range as a parallax approximate region. Aparallax image refers to an image representing shift amount of pixelsforming the same spot image in two different captured images capturedsimultaneously. In the example of the present embodiment, a parallaximage refers to an image representing a shift amount in the U-axisdirection between the spot image in the first captured image 14L thatcorresponds to the pixels and the corresponding pixels of the same spotimage in the second captured image 14R.

A description of the specific processing of the parallax approximateregion is now provided with reference to the example illustrated in FIG.5. FIG. 5 is a conceptual view of parallax d in a subject region 16illustrated in FIG. 4(1). A plurality of squares each surrounding anumber in FIG. 5 corresponds to the pixels in the subject region 16.Parallax d of each pixel is illustrated in the position of the pixel.The thick line in FIG. 5 corresponds to the pixels constituting anoptical image of a vehicle in FIG. 4(1). Parallax d of the pixelssurrounded by the thick line is generally within the range of 79 to 82.Parallax d changes depending on a distance from the stereo camera device1. A subject located at substantially the same distance from the stereocamera device 1 is subjected to substantially the same parallax d. Asubject with a measurable size on the first captured image 14L and onthe second captured image 14R is located at substantially the samedistance from the stereo camera device 1, and parallax d of the pixelsforming the optical images of the subject is within a predeterminedrange. In other words, an object is present in a region in whichparallax d is within a predetermined range. In the present embodiment,the controller 13 determines, as the parallax approximate region, aregion including continuous pixels in which parallax d is within apredetermined range.

The controller 13 extracts from the parallax approximate region at leastone first feature point P₁. The first feature point P₁ refers to acharacteristic point on the first captured image 14L, that is, a pointhaving a feature value of the pixel that satisfies predeterminedrequirements. For example, a vertex of edge at which a differential ofbrightness value is a predetermined value or more may be regarded as thefirst feature point P₁.

The controller 13 determines whether the first feature point P₁extracted from the first captured image 14L is suitable to be used forcalibration processing. When the first feature point P₁ is a part of atleast one of a linear edge and a repetitive pattern, an error tends tooccur in matching the second feature point P₂ with the first featurepoint P₁. For the above reason, the controller 13 determines whether thefirst feature point P₁ and a region including the vicinity of the firstfeature point P₁ include a linear pattern or a repetitive pattern. Thesecond feature point P₂ refers to a feature point on the second capturedimage 14R that has a feature value within a predetermined range of thefeature value of the first feature point P₁.

When it is determined that the first feature point P₁ is a part of atleast one of a linear edge and a repetitive pattern, this first featurepoint P₁ is not used in the subsequent processing, and another firstfeature point P₁ different from this first feature point P₁ is used.When it is determined that the first feature point P₁ is not a part ofat least one of a linear edge and a repetitive pattern, the controller13 retrieves the second feature point P₂ corresponding to this firstfeature point P₁ for extraction by using a conventionally knowntwo-dimensional pattern matching method. The controller 13 performs thetwo-dimensional pattern matching with sub-pixel precision by using forexample the interpolation method.

A starting point P_(S) of the retrieval of the second feature point P₂in the second captured image 14R is positioned at (u_(L2)+d, v_(L2)),which is offset by parallax d in the U-axis direction from the position(u_(L2), v_(L2)) same as the first feature point P₁. The controller 13retrieves the second feature point P₂ within a predetermined dimensionalrange centered around the position (u_(L2)+d, v_(L2)). The predetermineddimensional range may be the range within 1 to 2 pixels from thestarting point P_(S) in the U-axis direction and within 1 to 2 pixelsfrom the starting point P_(S) in the V-axis direction.

Based on the position of the first feature point P₁ and the position ofthe second feature point P₂ that have been extracted, the controller 13calibrates the first imaging unit 11L, with reference to the secondimaging unit 11R.

Here, a concrete example of the calibration performed by the controller13 is described.

In the following concrete example, a parameter corresponding totranslation of the first imaging unit 11L in a direction perpendicularto the baseline length and to the optical axis is set as an externalorientation parameter to be updated. The parameter corresponding totranslation in the direction perpendicular to the baseline length and tothe optical axis extends in the Y-axis direction in the spatialcoordinate system illustrated in FIG. 1. X-axis and Y-axis in thespatial coordinate system are respectively parallel to U-axis and V-axisin the image coordinate system.

In a case where the position of the first feature point P₁ extracted asabove is represented by (u_(L2), v_(L2)) and the position of the secondfeature point P₂ extracted as above is represented by (u_(R2), v_(R2)),the controller 13 determines that various positions of the firstcaptured image 14L are offset by the shift amount Δv=v_(L2)−v_(R2), withreference to the second captured image 14R. The controller 13 determinesthat the first imaging unit 11L is offset by the shift amount ΔY_(L) inthe spatial coordinate system that corresponds to the shift amount Δv inthe image coordinate system, with reference to the second captured image14R, and updates the position Y_(L) of the first imaging unit 11L in theY-axis direction by using the shift amount ΔY_(L). The controller 13performs the same processing when the pitch angle ϕ, which is rotationangle around X-axis, is set as the external orientation parameter to beupdated.

Next, processing performed by the image processor 10 according to thepresent embodiment is described with reference to the flowchartillustrated in FIG. 6. The controller 13 starts the processing uponreceiving at least one of a start-up instruction, a stop instruction,and a calibration control executing instruction from the stereo cameradevice 1.

Firstly, the input interface 12 receives input of the first capturedimage 14L and the second captured image 14R, which have beenrespectively generated by the first imaging unit 11L and the secondimaging unit 11R (Step S1).

Subsequently, based on the pixel values of the first captured image 14Land the pixel values of the second captured image 14R that have beeninputted to the input interface 12, the controller 13 calculatesparallax d of the pixels of the first captured image 14L with respect tothe corresponding pixels of the second captured image 14R according tothe one-dimensional matching (Step S2).

Subsequently, the controller 13 determines the parallax approximateregion in the first captured image 14L (Step S3). Then, the controller13 retrieves from the determined parallax approximate region at leastone first feature point P₁ for extraction (Step S4). The controller 13determines whether the extracted first feature point P₁ is suitable tobe used for calibration processing (Step S5). After that, the controller13 retrieves from the second captured image 14R the second feature pointP₂ corresponding to the first feature point P₁ that has been determinedas suitable to be used for calibration processing, for extraction (StepS6). The first feature point P₁ is determined as suitable to be used forcalibration processing when the first feature point P₁ is not a part ofat least one of a linear edge and a repetitive pattern.

Then, based on the first feature point P₁ and the second feature pointP₂, the controller 13 calibrates the first imaging unit 11L, withreference to the second imaging unit 11R (Step S7).

According to the image processing device of the present embodiment, thecontroller 13 calculates parallax d according to the one-dimensionalmatching and also extracts at least one first feature point P₁ from theregion including continuous pixels with deviation in parallax d within apredetermined range. Accordingly, compared with cases where thecontroller 13 retrieves the first feature point P₁ over the entireregion of the first captured image 14L according to the two-dimensionalmatching, the first imaging unit 11L is calibrated rapidly, withreference to the second imaging unit 11R. Since it is highly likely thatthe pixels constituting the parallax approximate region do not containnoise, the controller 13 is able to extract the first feature point P₁with accuracy. This allows the controller 13 to perform the calibrationwith accuracy.

According to the image processing device of the present embodiment, thesecond feature point P₂ is extracted based on the first feature point P₁that is different from any first feature point P₁ corresponding to apart of at least one of a linear edge and a repetitive pattern. Sincethe first feature point P₁ is neither a part of a linear edge nor a partof a repetitive pattern, a feature point similar to the first featurepoint P₁ is less likely to be included in the first captured image 14L.Similarly, a feature point similar to the first feature point P₁ is lesslikely to be included in the second captured image 14R which correspondsto the first captured image 14L, excluding the second feature point P₂which corresponds to the first feature point P₁. Accordingly, the secondfeature point P₂ which corresponds to the first feature point P₁, isdetermined with accuracy and without the second feature point P₂ beingmistaken for another feature point similar to the first feature pointP₁.

According to the image processing device of the present embodiment, thecontroller 13 determines the starting point P_(S) for thetwo-dimensional matching based on the position of the first featurepoint P₁ and parallax d at the position, and therefore, the secondfeature point P₂ is retrieved within a range in which the second featurepoint P₂ is very likely to be present. This allows the controller 13 toextract the second feature point P₂ rapidly.

According to the image processing device of the present embodiment,since the controller 13 performs the two-dimensional matching withsub-pixel precision, the calibration is performed even when the firstimaging unit 11L is offset by a shift amount of less than one pixel,with reference to the second imaging unit 11R. That is, the controller13 calibrates the first imaging unit 11L with high precision, withreference to the second imaging unit 11R.

Although the present embodiment has been described based on thedrawings, it is to be noted that a person skilled in the art may easilymake various changes and modifications according to the presentdisclosure. Such changes and modifications are therefore to beunderstood as included within the scope of the present disclosure.

For example, in the present embodiment, the controller 13 may use aplurality of first feature points P₁ and extract a plurality of secondfeature points P₂ corresponding to the plurality of first feature pointsP₁. In this case, the calibration is performed with high accuracy evenwhen the rho angle ω_(i) of the first imaging unit 11L is misalignedwith the rho angle ω_(R) of the second imaging unit 11R.

Although in the present embodiment the controller 13 calibrates thefirst imaging unit 11L, with reference to the second imaging unit 11R,the controller 13 may also calibrates the second imaging unit 11R, withreference to the first imaging unit 11L.

Although in the present embodiment the controller 13 extracts the secondfeature point P₂ based on the first feature point P₁ extracted from thefirst captured image 14L, the controller 13 may also extract the firstfeature point P₁ based on the second feature point P₂ extracted from thesecond captured image 14R.

Although in the present embodiment the stereo camera device 1 includesthe image processor 10, another device may include the image processor10, and the controller 13 of the stereo camera device 1 may performcontrol for calibration based on the first captured image 14L and thesecond captured image 14R that are inputted from the other device to theinput interface 12 via a communication network or the like.

REFERENCE SIGNS LIST

-   1 Stereo camera device-   10 Image processor-   11L First imaging unit-   11R Second imaging unit-   12 Input interface-   13 Controller-   14L First captured image-   14R Second captured image-   15 Vehicle-   16 Subject region

1. An image processing device, comprising: an input interface configuredto receive input of a first captured image and a second captured imagecaptured by a plurality of imaging units of a stereo camera; and acontroller configured to calculate parallax by performing aone-dimensional matching based on pixel values of the first capturedimage and pixel values of the second captured image, extract one or morefirst feature points from a region in the first captured image thatincludes continuous pixels having a difference in parallax betweenadjacent pixels which is within a predetermined range, extract one ormore second feature points corresponding respectively to the one or morefirst feature points by performing a two-dimensional matching betweenthe one or more first feature points and pixels in the second capturedimage, and calibrate at least one of the plurality of imaging unitsbased on positions of the one or more first feature points and positionsof the one or more second feature points.
 2. The image processing deviceof claim 1, wherein the controller is configured to perform thetwo-dimensional matching based on at least one first feature point inthe one or more first feature points that is different from the firstfeature points corresponding to a part of at least one of a linear edgeand a repetitive pattern.
 3. The image processing device of claim 1 or2, wherein the controller is configured to determine a starting pointfor the two-dimensional matching based on positions of the one or morefirst feature points and parallax at the positions.
 4. The imageprocessing device of claim 1, wherein the controller is configured toperform the two-dimensional matching with sub-pixel precision.
 5. Theimage processing device of claim 1, wherein the one or more firstfeature points extracted from the first captured image by the controllercomprise a plurality of first feature points, and the one or more secondfeature points extracted from the second captured image by thecontroller comprise a plurality of second feature points correspondingrespectively to the plurality of first feature points, and thecontroller is configured to calibrate at least one of the plurality ofimaging units based on positions of the plurality of first featurepoints and positions of the plurality of second feature pointscorresponding to the plurality of first feature points.
 6. The imageprocessing device of claim 5, wherein the controller is configured tocalibrate a posture of the stereo camera based on the positions of theplurality of first feature points and the positions of the plurality ofsecond feature points.
 7. A stereo camera device, comprising: aplurality of imaging units; and an image processing device including aninput interface configured to receive input of a first captured imageand a second captured image captured by the plurality of imaging units,and a controller configured to calculate parallax by performing aone-dimensional matching based on pixel values of the first capturedimage and pixel values of the second captured image, extract one or morefirst feature points from a region in the first captured image thatincludes continuous pixels having a difference in parallax which iswithin a predetermined range, extract one or more second feature pointscorresponding respectively to the one or more first feature points byperforming a two-dimensional matching between the one or more firstfeature points and pixels in the second captured image, and calibrate atleast one of the plurality of imaging units based on positions of theone or more first feature points and positions of the one or more secondfeature points.
 8. A vehicle, comprising a stereo camera device,including: a plurality of imaging units; and a stereo camera deviceincluding an input interface configured to receive input of a firstcaptured image and a second captured image captured by the plurality ofimaging units, and a controller configured to calculate parallax byperforming a one-dimensional matching based on pixel values of the firstcaptured image and pixel values of the second captured image, extractone or more first feature points from a region in the first capturedimage that includes continuous pixels having a difference in parallaxwhich is within a predetermined range, extract one or more secondfeature points corresponding respectively to the one or more firstfeature points by performing a two-dimensional matching between the oneor more first feature points and pixels in the second captured image,and calibrate at least one of the plurality of imaging units based onpositions of the one or more first feature points and positions of theone or more second feature points.
 9. An image processing method, inwhich a controller in an image processing device performs stepscomprising calculating parallax by performing a one-dimensional matchingbased on pixel values in a first captured image captured by a firstimaging unit in a plurality of imaging units and pixel values in asecond captured image captured by a second imaging unit in the pluralityof imaging units that is different from the first imaging unit;extracting one or more first feature points from a region in the firstcaptured image that includes continuous pixels having a difference inparallax which is within a predetermined range; extract one or moresecond feature points corresponding respectively to the one or morefirst feature points by performing a two-dimensional matching betweenthe one or more first feature points and pixels in the second capturedimage; and calibrating at least one of the plurality of imaging unitsbased on positions of the one or more first feature points and positionsof the one or more second feature points.